SDI Effectiveness from the User Perspective

Definitions of spatial data infrastructure (SDI), and claims about its value, emphasise applications that solve real societal issues as the ultimate benefit. While it may be premature to expect that those benefits show up in many SDIs (if in any, by now), preparing for systematic evaluation is timely. Initiatives like the Spatial Data Interest Community on Monitoring and Reporting, in conjunction with the implementation of the Infrastructure for Spatial Information in Europe (INSPIRE) and the project Multi-view Framework to Assess National Spatial Data Infrastructures (NSDIs) funded by the Dutch innovation program Space for Geo-Information, are contributing to this goal. Drawing mainly from the information systems literature, we discuss possible measures of SDI effectiveness and present empirical results of a national survey of inter-organisational geographic information systems (GIS) in the United States. We model the measures of GIS effectiveness and other user-based perceptions of benefits against factors that tell us about possible determinants of inter-organisational GIS success that are potentially applicable to SDIs as well. Understanding the significance of those factors could help us move towards more useful and effective SDIs. Chapter 14. SDI Effectiveness from the User Perspective 274 14.1 USER PERSPECTIVE ON SDI Most definitions of spatial data infrastructure (SDI) include the notion of geographic information or database sharing, access and use (Groot and McLaughlin, 2000). Spatial data discovery and access are the necessary initial steps in SDI use which are facilitated through metadata catalogues (Craglia and Masser, 2002; Smith et al., 2004; Craig, 2005) and rely on metadata standards (Kim, 1999). However, the true demonstration of the value of an SDI (and return on substantial investment made world wide, as per Onsrud et al., 2004) is in its support in responding to perennial problems of poverty, disasters, urbanisation, healthcare, utilities and the environment, among others (Williamson, 2003; Masser, 2005). In Craglia and Johnston’s (2004) words: Many of the main challenges of contemporary society, such as protecting the environment, increased security, better transport, ‘socially just’ or ‘sustainable’ development, and enhanced services to citizens, require decision-makers to identify where need is greatest. To effectively target intervention, monitor outcomes, and assess impacts, access to geographic information (GI) is crucial. Ideally it should be easy to identify who owns GI, whether it is fit for the purpose in hand, how it can be accessed and integrated with other information. (p. 17) SDI evaluations that would tell us if it meets these expectations are still sporadic. Several studies discuss geo-portals as gateways to SDI. For example, Bernard et al. (2005), Maguire and Longley (2005), and Tait (2005) explore the capabilities of the second generation geoportals in order to access spatial data and services. The most common measure of access is by counting portal hits, as in Tait’s (2005) assessment of the Geography Network that receives 300,000 hits by estimated 50,000 users per day. Preliminary research on SDI use suggests that contemporary SDIs do not fulfill their purpose and expectations. Nedović-Budić et al. (2004), in their evaluation of the use of SDIs in the context of local planning in Victoria, Australia and Illinois, USA, conclude that state level SDIs do not effectively serve local needs. Based on Harvey and Tulloch’s (2006) study in 2002, most of local governments in the United States are either unaware of or do not take the SDI concept as relevant to them. A few studies go into the context and reasons for the performance of the SDI, such as Tulloch and Fuld’s (2001) finding about the A Multi-View Framework to Assess SDIs 275 technical challenges of various data formats and accuracy levels and institutional challenges due to data producers’ unwillingness to share data. These results are reinforced by a national scale survey conducted by Knaap and Nedović-Budić (2003) who find that integration and use of regional spatial data in an SDI environment is difficult. NedovićBudić et al. (2004) trace the insufficient utility of Victoria and Illinois SDIs to their inadequate contents and data scales and suggest the importance of the institutional setup of data collection and production. Van Loenen and De Jong (2007) also emphasise the role of institutions in the realisation of SDIs. Crompvoets et al. (2004) report that an unfriendly user interface and the discipline-specific nature of metadata and clearinghouses are among the primary reasons for the slightly declining trend of using clearinghouses. Askew et al. (2005) and Beaumont et al. (2005) share the UK’s experience in building an SDI on the government’s Information and Communication Technology (ICT) investments and assert the difficulties in developing geo-portal related partnerships due to varying levels of technological experience and differing goals and expectations among the partners. Interestingly, similar obstacles are encountered in the Asian context. Georgiadou et al. (2005) attribute the slow progress of Indian SDIs to its data-centric implementation. Puri (2006) adds the differing perceptions and expectations of various stakeholder groups as a reason for the delayed progress. Similarly, the deficiency in requisite human skills and technical infrastructure of participating agencies disrupt the Nepalese efforts to develop SDIs (Budhathoki and Chhatkuli, 2003). When summarising the experience with SDIs worldwide, Masser (2005) alerts that “some formidable challenges lie ahead and the task of sustaining the momentum that has been built up in creating SDIs in recent years will not be easy” (p. 273). These challenges require a critical examination of the purpose and application of SDIs. The ongoing evaluation research is still more concerned with access to spatial data than with use and utility of the infrastructure. There is no clear evidence about who are the users, what they use the information for and how well they are served by geo-portals (Askew et al., 2005). Access is a necessary but insufficient condition for ‘use,’ and the ‘use’ which follows access, determines the success of an SDI. Unfortunately SDI research to date has neglected the use and user aspects, and only recently have researchers started to call for usercentred SDIs (Williamson, 2003; Masser, 2005; Puri, 2006; Rajabifard et al., 2006; Budhathoki and Nedović-Budić, 2007). Chapter 14. SDI Effectiveness from the User Perspective 276 This chapter focuses on SDI effectiveness by first reviewing relevant conceptual developments and illustrating them by presenting US-based empirical assessment of the benefits derived from interorganisational sharing of geographic information. The chapter complements the ongoing efforts in establishing SDI evaluation frameworks and measures (Craglia and Nowak, 2006; Grus et al., 2006; Lance et al., 2006). 14.2 CONSTRUCTING THE AFFECTIVE USER The term ‘user study’ is focused on the ‘user’ and not the ‘system,’ first appeared in the field of information science in 1965 (Wilson, 1994), yet only becomes established in the 1980s (Dervin and Nilan, 1986; Wilson, 2000). The fundamental premises of these studies are that: a) better service to the user increases the utility of information systems; b) user needs and uses are the focal point of information system development; c) there needs to be openness to redesigning the system in order to more effectively serve users; d) capitalisation on technological advances ought to be used for the benefit of users; and e) system designers are accountable to users. Based on the unit of analysis ─ individual or organisational user ─ and on the intensity of their information seeking behaviour and information use ─ passive or active ─ we identify the following four basic user types: individual as passive recipient of information, individual as active information agent, organisation as passive information user and organisation as active information agent. There is also a gradation of types between these four basic categories and several other criteria, such as social units (group, community, etc.), demographic characteristics (sex, age, profession, etc.) and mode of use (home, office, etc.) that could be used to modify the types. We cogitate that both individual and organisational users who passively receive and use information are unlikely to engage in the system building process. Consequently, such users exert little influence on the development of an information system and its applications. On the other hand, users who play the role of information actors and provide insights and ideas to designers actually engage in the process of designing and building of information systems. This is the mode where individual and organisational learning comes into play. 2 There are other possible categorisations of users; for example, based on the extent to which data is required for execution of the tasks; the extent to which data is being A Multi-View Framework to Assess SDIs 277 Previous research suggests that learning and information use are connected. Both individuals and organisations with minor (or so called ‘single-loop’) learning characteristics are likely to make less use of information (Argyris, 1976; 1999). On the other hand, if users are more engaged (or ‘double-loop’ learners), that is if they question designers’ assumptions about the information system, they get the designers to probe their real information needs and uses leading toward more useful and effective systems. A system which addresses the user’s information needs also advances their learning. In reality, very few users are ‘double-loop learners’ and, therefore, it is often difficult to establish the relationship between users and designers. Different strategies need to be deployed in the design, implementation and use of information systems to fit each of the user categories and to achieve effective use of the systems. In information system building, including SDI developments, it is often too easy to adopt a standardised approach regardless of the type of user or user organisation, even though individualised and contextualised approaches are called for. Therefore exploring intended users and uses before the actual system building would lead to potentially more useful systems. The users should be prepared to be involved if the design is to benefit from their action, and if they are to benefit from the particular design. A useful model in understanding the use (or non-use) of innovation is concern-based adoption model (CBAM) (Newhouse, 2001). The CBAM consists of three dimensions: stages of concern, levels of use and innovation configuration. Concern in CBAM, according to Hord et al. (1987), is defined as feelings, reactions and thoughts that the users have about the innovation that affects their life (Table 14.1). They group these concerns into three categories: self task and impact. Knowing the users’ concerns is vital to designing effective interventions by aiding their movement through the stages of these concerns. For instance, if intended users are unaware of the innovation, an improved user interface will not generate much impact. Similarly, if the concern of the majority of intended users is in the ‘self’ stage, it is useless to assess the impact of an innovation. used in line with the purpose it was collected for; use vs. re-use of information; distinction between public, private and non-profit sectors, academic institutions, and individual citizens as users; direct and indirect users ─ the latter using information prepared by others; etc. Chapter 14. SDI Effectiveness from the User Perspective 278 Another important dimension of CBAM is the level of use. Hord et al. (1987) dissect the use of an innovation as: non-use, orientation (learning), preparation (plans to begin), mechanical use, routine (established pattern), refinement (introducing changes), integration (coordination with other users), and renewal (seeking alternative uses). The authors note that we often seek to measure effectiveness of an innovation without ever examining its use. They suggest deferring the evaluation of outcomes if a significant percentage of users have not reached the level defined as routine use. Finally, CBAM also anticipates that at least some part of the innovation will be used by various users differently from the designer’s intentions. Innovation configuration means that the users appropriate certain components of innovation. Therefore, knowledge about the configuration of an innovation is also important for measuring the level of use, and ultimately, the effectiveness of the innovation. Table 14.1: Typical Expressions of Concerns about an Innovation (After Hord et al., 1987; Loucks-Horsley, 1996) Stages of Concern Expressions of Concern 6. Refocusing (I) I have some idea about something that would work even better 5. Collaboration (I) How can I relate what I am doing to what others are doing? 4. Consequences (I) How is my use affecting learners? How can I refine it to have more impact? 3. Management (T) I seem to be spending all my time getting materials ready 2. Personal (S) How will using it affect me? 1. Informational (S) I would like to learn more about it 0. Awareness (S) I am not concerned about it Category: S = self, T = task, I = impact The user typology presented above is congruent with CBAM. Passive users are likely to remain (or spend more time) at lower levels of use and are unlikely to engage in configuring a system. Their concerns will also likely be limited to ‘self’ and ‘task’ stages. Higher level users are more likely to be active, to pursue the so-called ‘double-loop’ learning and generate impact. User concerns and the level of use of an innovation, however, do not exist in a vacuum. Information needs and use goals arise from the users’ situational gap that leads to information needs (Dervin and Nilan, 1986). Accordingly, a (mis)match between users’ information needs and the capabilities of an information system affect the user’s use of the information system. Further, the way the user engages in seeking and using information system depends on whether it is an A Multi-View Framework to Assess SDIs 279 individual or organisation, an active or passive information user, along with other cognitive factors. Complementing this recognition is a trend of viewing users as innovators, ‘sense makers’ and ‘domesticators’ of information technologies and systems (Dervin, 1989; Williams, 1997; Bruce and Hogan, 1998; Griffith, 1999; Stewart and Williams, 2005). The central tenet of the domestication and its associated concept of idealisation-realisation technology (Bruce, 1993) is that technology gets appropriated and its meaning is constructed by where the use is situated. By implication, the designers can not design the system and can only invoke the design process. It is through the users’ continued appropriation that an information system and services become useful. This process is also likely to underlie the effectiveness of existing and prospective SDIs. 14.3 CONSTRUCTING AND MEASURING EFFECTIVENESS OF INFORMATION We review the construct of information effectiveness from several disciplinary streams: ICT, conceptualisations, Information Systems (IS) literature, geographic/land information systems (GIS/LIS) and the recent SDI works. 14.3.1 ICT Definitions From the broad ICT perspective Blomberg et al. (1994) define usability as “the general intelligibility of systems, particularly at the interface” (p. 190). From usability they differentiate usefulness, which “means that a system’s functionality actually makes sense and adds value in relation to a particular work setting” (p. 190). The concept of effective use subsumes both usability and usefulness. The effective use of ICTs, according to Gurstein (2003), is “the capacity and opportunity to successfully integrate these technologies to achieve the users’ selfor collaboratively-defined goals.” We argue that the effective use of ICT requires: carriage facilities (that is, appropriate communication infrastructure), input/output devices, tools and supports, content services, service access/provision, social facilitation (for example, network, leadership and training) and governance (Table 14.2). Chapter 14. SDI Effectiveness from the User Perspective 280 Table 14.2: Conditions of effective ICT use (After Gurstein, 2003) Carriage Facilities Appropriate communication infrastructure (eg. broadband, dial-up, WiFi, satellite ) to carry the task at hand Input/Output Devices Physical devices such as computers, PDAs, printers Tools and Supports Software, physical supports, protocols, service supports etc. Content Services Content that is relevant and suitable to solve particular problems. Some of the content related issues are language, design, literacy level, localisation of references, links etc. Service Access/Provision Organisational infrastructure, locally available support service Social Facilitation Social infrastructure such as conducive social network, local leadership, training Governance Financial, policy or regulatory regime 14.3.2 Information System Definitions In the Information System (IS) realm, DeLone and McLean (1992) suggest the amount and duration of use (for examples, the number of functions performed, reports generated, charges and frequency of access), and nature and level of use, are objective measures of information system use. Questions about who uses the system, motivations and willingness for use, and the purpose and nature of system use are also relevant. Measuring the effect of an information system, in terms of information use, assumes that the effect of a technology is proportionate to its use in supporting organisational functions, tasks, and projects. Moving towards the information impact, the authors reference it with respect to individual and organisational effects. The former are recognised if information influence decisions and the latter are manifested through improvements of organisational performance and better understanding of the decision context. Information systems may influence the behaviour of individuals who rely on these systems for information, expertise and decisions. A good system may improve individual performance by facilitating better understanding of the decision factors and by increasing decision-makers’ productivity and confidence (DeLone and McLean, 1992). A good system may also affect the course of action taken by decision-makers and ultimately change their perceptions of the value of a particular information system, althouh capturing decision-making and decision support effects is challenging. Efficiency and effectiveness are the criteria commonly used to evaluate how information systems affect organisational performance (Mundel, 1983). Efficiency is defined as the ratio of outputs to inputs. Efficient systems either minimise the use of the financial, staff, space, A Multi-View Framework to Assess SDIs 281 and time resources needed to produce the same level of output, or increase productivity using the same level of input. Therefore, efficiency can be expressed as cost savings, cost avoidance, or productivity gains. Beyond the efficiency, system effectiveness means generating a product of better quality or accomplishing an intended purpose. Heffron (1989) notes the multi-dimensionality and difficulty of measuring the concept of effectiveness. Organisational effects are frequently evaluated using cost-benefit analysis, despite its limited utility. Alternative evaluative approaches, such as cost-effectiveness, conjoint (value) analysis and measures of organisational goal achievement and productivity, are devised to deal with the shortcomings of the traditional cost-benefit method (Nedović-Budić, 1999). Information science also considers user satisfaction as one of the most prominent measures of system performance and effectiveness. User satisfaction has to do with what users want, as opposed to usefulness which relates to their true functional needs. In fact, Gelderman (1998) provides evidence that user satisfaction is the “most appropriate measure for information system success available” (p. 11). User satisfaction affects the achievement of information system goals, employees’ quality of work life and system’s use (Torkzadeh and Doll, 1991). In the decades-long attempts to model information system success, DeLone and McLean (1992; 2003) define user satisfaction as a way in which those who receive information react to the output generated by an information system. Management information system researchers have also asserted the superiority of the user satisfaction approach over objective measures of system use and performance (Raymond, 1987). The premise of the user satisfaction approach is that “a ‘good’ information system perceived by its users as a ‘poor’ system is a poor system” (Ives et al., 1983, p. 786). The authors advocate employing user information satisfaction as a surrogate measure of information systems’ utility in decision making. Examples of other measurement approaches include: Torkzadeh and Doll’s (1991) differentiation between general and satisfaction with specific applications; Baroudi and Orlikowski’s (1988) psychometric evaluation of electronic dataprocessing staff and services and of user knowledge and involvement; and Davis’ (1989) correlation of perceived usefulness and perceived ease of use with the actual system use. User satisfaction with the information, and ultimately the system’s implementation success, are affected by a variety of factors, such as user attitudes (DeLone and Chapter 14. SDI Effectiveness from the User Perspective 282 McLean, 1992), differences between the users’ and designers’ cognition (Griffith and Northcraft, 1996), as well as the mutual and individual expectations regarding the technology that is introduced. Finally, a comprehensive approach to information system performance is offered by Chang and King (2005). They devise an Information Systems Functional Scorecard (ISFS) and validate its components using factor analysis and structural equation modelling. The ISFS relates to the business process and organisational performance with three dimensions and eighteen uni-dimensional factors. The dimensions include: system performance, information effectiveness and service performance. System performance addresses the technical aspects, and direct impacts, on work of any information system used by the regular organisational users; service performance deals with quality and flexibility of services; information effectiveness “[a]assesses the quality of information in terms of the design, operation, use and value provided by information as well as the effects of the information on the user's job” (p. 90). Factors of information effectiveness are: intrinsic quality, reliability, contextual quality, presentational quality, accessibility, flexibility and the usefulness of information. 14.3.3 GIS/LIS Definitions GIS/LIS research also offers some suggestions toward the evaluation framework and its criteria. For instance, borrowing from Jordan and Sutherland’s (1979) program evaluation framework for assessing public expenditures on land information systems, Clapp et al. (1989) identifies: a) operational effectiveness, which considers program outputs and includes information availability, and public and private understanding; and (b) program effectiveness, which focuses on program effects such as enhanced decision making and timely problem recognition. These program and decision-making dimensions of effectiveness primarily represent the individual effects. Antenucci et al. (1991) distinguish between five types of GIS benefits: (1) quantifiable efficiencies and improvements in existing practices; (2) quantifiable expanded or added capabilities; (3) quantifiable unpredictable events; (4) intangible benefits and advantages and (5) quantifiable sales of information and resulting service benefits. They also classify benefits as direct and indirect. Direct benefits accrue to the organisation or unit sponsoring the GIS; indirect benefits accrue to other individuals and agencies. Initial benefits often accrue as organisational efficiencies and may also A Multi-View Framework to Assess SDIs 283 result in the generation of revenue (Huxhold, 1991). For evaluating a GIS in a local government setting, Worrall (1994) adds to the list of efficiency-related benefits: better service; improved regulatory functions; more accurate referencing of property, land, and infrastructure along with improved consistency. According to Antenucci et al.(1991), the benefits of GIS effectiveness can occur at the operational, management or strategic level. Calkins and Obermeyer (1991) offer a taxonomy of use and value of geographic information that is also intended to stimulate the further development of evaluation methods. They loosely group twenty-four issues into six categories of relevant questions to ask about the use and value of geographic information. These categories attend to: the use of geographic information; the effectiveness and benefits of geographic information use; the measurement of benefits associated with information use; the characteristics of geographic data and spatial analysis and the characteristics of the organisation. The basic premise of individual effects is that information has value only if it is used and influences decisions. Improved decisions, in turn, assume that the content and amount of supporting information are changed and that value is added to it. Tulloch (1999) goes one step further in systematically defining a set of constructs that can be used in the assessment of GIS technology. Building on their study of modernising land records and developing Multipurpose Land Information Systems (MPLIS) in local governments, the authors propose efficiency, effectiveness and equity as the criteria for determining the usefulness of an MPLIS. Efficiency benefits are expected at the record-keeping stage of MPLIS development and usually occur within a single agency. Effectiveness benefits accrue across local government agencies once a MPLIS is used for analysis. Finally, equity is achieved during the democratisation stage, when the benefits from an MPLIS are distributed throughout the community. Zwart (1991), who also focuses on decision making, proposes a method that relies on two criteria that can be applied to measuring the value of land information systems. These criteria are the degree to which information generated by a land information system is used and the level of importance of decisions affected by such information. With respect to the degree of use, information is classified as: not even referred to, used to support values or decisions, or used to change values or decisions; with respect to the level of importance the Chapter 14. SDI Effectiveness from the User Perspective 284 decisions are classified as: important or not so important. Agumya and Hunter (1996) attempt to enhance this method by introducing the measurement of risk associated with the use of information that is of uncertain quality. They define risk as “the probability that an adverse event will result from a decision, multiplied by the cost of that event” (p. 349). 14.3.4 SDI Definitions SDI research has only begun to address the evaluation issues. Georgiadou et al. (2006a) apply Clement and Shade’s (2000) ‘rainbow’ metaphor to SDI. The metaphor includes the following elements: carriage, devices, software, content, provision, literacy and governance. With some differences, this metaphor substantially overlaps with Gurstein’s (2003) framework and derives from the same source. The authors associate each element with a set of policy questions. Governance, in particular, brings about the issues of community involvement in decision-making and impact assessment. Georgiadou et al. (2006b) also suggest a variety of methodologically rigorous evaluation approaches suited to progressively complex focus on data, services and E-governance. Again, the reference to governance bears direct relevance to the decision-making process and leads us to the individual and often, by implication, the organisational effectiveness of information. Among the sporadic empirical works, a study by Lance et al. (2006) reviews evaluation activities of practitioners involved in SDI developments. The authors find that the practitioners favor the ‘control’ evaluation method for assessing their success. This method is quantitative in nature, and is primarily focused on examining the efficiency and rationality of investment decisions. The authors rely on the concepts of ‘timing’, ‘perspective’, ‘formal demand’, ‘use’, and ‘input specificity’ and discover that “the most comprehensive 3 Before estimating how a GIS (or SDI) affects decisions, it is important to understand the ways information is used in the decision-making process. Dickinson (1990) reviews methods for modelling the decision process, including critical path and Program Evaluation and Review Technique (PERT) methods, data flow diagrams, decision trees, entity-relationship diagrams, flow charts, Markov chains and Petri Nets. Petri Nets method is suggested for applying to the analysis of decisions that are based on spatial information. Nets can represent complex decision-making systems that consist of asynchronous and concurrent subsystems and are therefore particularly suitable for representing the flow of control and information in organisational systems. A Multi-View Framework to Assess SDIs 285 practices have resulted when ‘control’ evaluations have been in compliance with a demand from an executive agency, such as a central budget agency, and when there has been specificity of inputs” (p. 65). Another empirical study of socio-economic impacts of Catalonian SDI (Craglia et al., 2008), identifies significant efficiency benefits at the level of local public administration and effectiveness benefits accrued to the public and to companies dealing with public administration. The authors employ a theoretically grounded framework on benefits from e-government services developed by the European Commission’s e-Government Economics Project (eGEP, Codagnone et al., 2006). Out of the 90 proposed indicators, the Catalonian study uses a subset that is suited to the local context. In exploring the wider socio-economic impacts in qualitative terms, the study shows that smaller local authorities are the key beneficiaries of web-based spatial services which are narrowing their digital divide with larger ones.